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Please replace ihe specification withtlie following substitute specification showing 
changes made via strikcthrough and imderlining: 



Received from < 70361 08668 > at 6/30103 5:35:57 PM [Eastern Daylight Time] 



2 



c 



7036108668 



MAIL ROOM Fj 



17:12:49 06-30-2003 



4/54 



Docket No. G2622-908496 
Appl. No. misi^iii 

Preliminary Amendmenl IIM wUh RCE Request 

SYSTEM AND MEfHOD FOR INDEXINC RECORDlMGS OF OBSERVED TKST 



EVjENIS 



FI¥MS 



This application is a continuation in part of provisional patent application serial no. 
60/127,194 on March3U 1999, the disclosure of which is herein incorporated by reference. 

rhis invention was supported made with a grant from the U.S. government which may 
have rights to the invention. 

COrYRiGf lT NOT1FICA7 4QN 

Portionfi of th i n pat e nt oppUootio ft' ^onUiin materials that aro subj e ct to copyrigl H 
protection. Th e copyright own e r has no objce t ion to the facsimii e r e production by anyone 
the patent docum^^nt or pat e nt disclosur e , as i t-appoarsH £i the Pat^tnl and lYademark Office 



FIFJ.D OF THE INVENTION 

afifiefiRm e nt m e thods Qt>d o peratjo ns, including th e training and mon i toring of l m < 




r e trieval of r e cord i ngr i of interv i ew ^-o r observed events and, mor e part i cularly, to ft frystea 

combining nyr»t e matic use by on obg efv cf of pro defined d e scriptiv e or quantitative 
m e ofiurem e nt it e ms with -simuk^i ^us digital recording to automatically genorato -ifid^^fceik 

The present in vention gen erally rehu cii to information processing systems. More 
p^icularlv. the invention i."^ a d ^Ui caPtMre and access system that correlates data with 
r^eord^d sif^nals wt jJijt^inde3i.bja;aC:d..0n.p^ ^^^^^I'ffg^lglill^ time of Cjapturc but 

allciw<ijaicr .annotation Or additional ind exing without aftcctinf; lhfc...Qm;inul.index identifiers 
or.lhc recorded sign als, f he invention speci fically pri- tyidcs a method, for indexing of digital 
recordings of events bv use of pre-entercd or„ t>re-exislinfi measurement items that serve a<i 
index information desif>nato i^.aeh.a5„lvCV. vvords or hibels. Such a method is p articularly 
»5.eMm tho$c u^ed in a systematic inte rview or assessment method . 
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4 wMany events jn the medical, legal , scienlific , biisinessT and education domains, 
among others, it is neoessa f^y4 t> systomaticaliy obiierv e and d e scrib e , ariri e fi.q, rat e , or oth e rwi se 
measur e e v e nts or proc e sses (k ^r, phen o mena). It in also us e fu l to mak e r e oordings of th e s e 
e v e ntf i and to b e able to -pl^ back ihcae rccordingti. For e xampl e , physicianr i may ur» 6 audioy 
video, or e l e ctrophys i olog ic al recording of an interview^ a phya i oa! e xamination, a diagnost ic 
proc e dur e , an oporatk> ft;f Qr autQpav. iQda Y are bein^ digitally recorded, PhYsiciaris may record 
a p hysical examination, operation^ or autORiiy and-i»ay need to retri e v e and e xamin e spc eifio 
parts o t annotate t})at recording at some later time for various purposes. Attorneys may -eaHy 
eut-andto notes, obge^ or other conte nt in order to make a more cpmplete record or 

draw previously unseen trends. Lu^yyeni may seek to record a witness deposition and later 
playback s e ct i ons otl mk that oral testimony in ord e r to link th e t e stimony to oth e r tQ evidence 
(documentary, video, or testimony by other witnesses) in order to draw parallels?, or highlight 
inconsistencies , or for oth e r purpos e s. S o^akieientisls-a pply systematic obser>^ation and 
rating methods to study soci e ty and cultu re. The frequency of such recordings, audio, video, 
or other, is likely to increase as their usefulness continues to grow ^ general, the usefuln e ss 

playback fun etio ns, and how accurat e ly and e tTici e ntly r e trieval and - play back can be carri e d 
out; Pufpose »-in&itid ^Jhexfeby.at{o.w.int^ fQr r eview of the assessor's conclusion or diatmoses^ 
providing an educational or teaching tool to illustrate obs e rvation, as s o G Stn entr'afid 

r e tri e val of m uteipte- interviows, events, or s e gm e nts of i nt e rv ie ws o^ even i s for comparison 
and other study: - r eviewiflfrai h)Dtimum assessment interview techniques, d ocumcnlinf;^ the 
interview for legal and/or other purposes ; and oth e r sim il ar and - varied u s es in mcmy domains . 

As a more detailed and prototypica l e xampl e , medioi ne -Me.dieine and public health 
are -<lomains in which considerable effort has been devoted to the development of tools (e.g., 
questionnair e s or assessment schedules aml instruments^ for conducting .syst e matic 
assessment intervicws-and-ex-anHnati ons. A main .^The objective of these efforts has often 
been to improve the aeewaoyrreliabiliiy^ and efficiency of a clinician's diagnosis of ahis/her 
patient. 

Among medical disciplines, psychiatry researchers have perhaps allocated the most 
efforts to developing patient asse.ssment m:etibods and tools aids. Examples of the types of 
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iUisessm e nt instruments that hav e b e en dev e loped, \ ^4^i ch arc composod ot d e sGriptor or 
quantiUlivc (it;so t iiiinricnl (i.e., measiiir e m e nt) it e m.? or qu e^^fl ^inslrumcnts develop ed include 
the Sln^ctured Clinical Interview for DSM (SCID), the Diagnostic Interview Schedule (DIS), 
the Composite International Diagnostic Interview (CIDI), the World Health Organization 
(WHO) International Personality Disorders Examination (IPDE), and the WHO Schedules for 
Clinical Assessment in Neuropsychiatry (SCAN), These tests have been standardized around 
aots of inquiries or m e asur e m e nt itemri. MeaQU fe mcnt items topically ar e oompo s ed -efe-t) 
definition of a phenom e non of int e r e fit; 2) instructions regard i ng obsen^ation of the 
phenom e non; and 3) a quantitativ e rating s eal e^ The scient i tlc fi e ld of m e aqurem e nty known 
as psychom e trics, includ e s ntudy of and meth ods-pe rtaining to syst e matic and gtan dQjr-diiEc4 

importance include efficiency, r e liabil i ty (conr i is te ncy). and validity faccuracyj. a core set of 
inquiries, or.directions. 

The response to or rating of standardized questions , inquiries, or m e asur e m e nt itcna^ 
an44ni$m»i^k»ns . or directions is of interest for the assessor, both in terms of the assessment 
fesutei suhstantive response as well as the behavior manifested in the delivery and,of the 

may want to measure-ef-fate4h<5 behavioral responses of patients ^ or m ay wish to observe-and 
fate the actions of other-^psychiairists in training. Cardiologists may want to meiisure physical 
responses to changes in applied stress, such as chang e s int he EKG readings during a cardiac 
stress test. L^awyers may seek to identify inconsistencies between witnesses to an accident 
using standardized questions. Similar examples are found in virtually every profession where 
a pattern of sygt e matio -obse rvation and testing for response? and analysis of thHtthe response 



arip e ctn of obscrv ed-^ vonts are u s ed to assist in th e s e proces ses J^ii^ . 
However, most-anyone who has participated in fetfieving^and- 



these interview or testing fee- 
retrieval-i 




iew-and-sk>dyrceords realizes that the 



efllciem-.vy.ay to. index the video, audio or ot her type of record ing when the interview was 
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recQjrded. much timg-feOulJ be saved when later seap^Jbinjj:. the records tor particular 
intormation^ 

Ft is. therefore, ai i..o.bigct of the nresent inventi on to provid e a system and method for 
conducting, recording, P^y'"K b^*^ editing or annof 4tin]g.diffltal recordings in audjo^ 
video, or o ther format.. 

Key to any usef ul recprding system based on digital metii'a is the ability to efficiently 
generate meaningful indexes for the recorded matcriaK Such indexes would be ospoc 4atty 
valuabl e for s trii ^tufed or tiyiit e matic i nformat i on -ga thering proc e sses i n wh i ch profo 86ioftat-ef 
^5e*e*^€^»tain darda arc r e quir e d and/or i n wh j^h considerab le e ffortr . ar e made i fV'tfakHftg^ 

tist-Bf^es^ons Hiivina a pati^t ^ u-e well suited tor structured infomiation gatherjn^ -e^^ 
that usg a sy stematic q u c;sUQn-and-answer method 

containi ng a $et or IMM Standardized inquiries, e.p.^ a list of television interview questions^ a 
health questionnaire, a clinical or research medical assessment, conducting a set of 
depositionr^tfiffi g Questions, a list of actions to activate brain function in different ways, 
conducting focus groups, e thnogroph i c research, or administ e rinfi aod a graded cardiac stress 
test, among others. 

In the fiekfa rfiejd of medicine and health, the established means of recording 
structured assessments has generally been by use of analog-audio or video tape recorders. 
Indexing, if used at all, has been g e n e rally through moans such as length counters and 
manually added entries , or w i th hybrid oom bt nations of tap e record e rs with mtofOprocoatsQr 
syr»tems; Thi s makes . . This made the process-e#<^ ating ind e x e s tedious and often incomplete, 
hi addition, even aftor ind e x e s are cr e at e d, the use of tape recorders means that the process of 
searching and playback r e mains is slow and incHicicn l. Finally, cr e ation of geno raliy 

In the field of digital recording, many indexing schemes have been developed, 
ranging from low-level encoding to linking of text with video frames. Industry standard 
digital encoding methods such as the MPEG-2 (Society of Motion Picture Engineers) format, 
include channels for indexing based upon time code. Mowever, most of these approaches 
generate index entries in a form not directly useful for retriovalr-playback^ and management, 
buir^^hcr provide basic mechanisms tor supporting indexing. For e xample^ o n e prior art 

One system is known that uses computer voice-to-text processing to automatically 
generate words that are then used to index the seuf^ecKligital audio recording. Although 
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soffl^KHieu^eiul* it does not provide a more organized Icvci of indexing, since many words 
will be used in many areas of a recording, and hence as such do not provide specific indexing 
to unique areas or segments of a recording. For indexing to be more useful^ the ind e xef i or 
index items should contain meaningful and unique information that is iissociated with 
tffliOHOlhfe recording locations or r» e ftmentS y"MaiH dja<daU(?n..Or..se^m Any indexing 
method mielhod. am support this, but it is gefterall^y^iedious and inefficient to generate such 
index items manual ly. 

E?iamples of prior art ind e xing gyete ffls include* U.S. Pat. No 1,84 1,3 8 7, whioh 

on- top of th^:g touchpad. This syst e m ia simp ly'^nFibe ans for replay of record e d audio o r-videe 

numb e r, nor do e r i it prov ld^ ny other means of id e ntif>1ng^ r ie arohing^ -e f retrieving r e cord e d 
mat e ria l s, nor do es it r i upp Of^ syehomt?lric and mor e g e n e ral obqerv atie n^ moasuromtint, 
retrieval^ train i ng, monitoring, or comparison functions and op e rationffi In tt H>-'p rofessional 
and technical ar e nas, it is gyflt e mati o- as well as quick acc e ss to rip e c i t i o pa ssag es or segm e nts 
of fccordud mat e rial that d e term i nes a s yst em's usefuln e ss, wh i ch i s not adequa te^y- prov^d e d 
by utilizing th e position of handwritte n-^^o tcs for ind e xing. 

European pat e nt appl i cat i on publ ic ation EP 0 A95 612 disclos e s an indexi ng-systom 
that eomnjcts manually e nt e r e d compu teF noles lo audio or v i d e o r e cordings, a nd ^ermits th e 
us er to retriev e and p l ay s e ct i ons of a ro ee rding by s e l e cting a not e . Tli e indexin g4s-bas6d 
upon spatia l location and orgat>i?a tte n - of a set of not e s with i n a document fat her than th e 

within the e l e ctronic docu meftty-BeGa use of this limitat i on, wh en-a-user-wish 

recording, since th e lat e r not e s will be time stamped with th e tim e they wore entered rath e r 
automatic ind e x linkag e-t e the earlier material. As w i th U.Sv Pa t; No; ^,811^3 8 7, this syst e m 
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fatwgH»^bsefV^ ev€nts, and doea not provid e a m e ans of -fC»t ricval and playback of record e d 

e ^trre i ating previously recorded vid e o mat e rial w i th a s tonditrd text transcript for th e purpos e 
^"%m^ Ming an index that can ihon b e iia e d by riottworo that search ei j t e xt and finds 
associated video indexes, 'I hia syat e m han no copao h y - to open^lc t in real tim e situationn. Tt 
does^t-provide-m ^nti for automat e d d e finition of se gments-of 
pfovi^i^-afHiu tomatcd m 

only ba rved-iipofv-opc-rato r - cnicrcd key word comm e ntrs and -is-i 
the-mfefmat}efHn-th c original text. It vj tiubj e ot to th e prob tc m of specific comment words 

ind e x e n to k ee p track of and e xe fe»£t o judgcmont about th e ohoio e ot'comme nt- words> a 
significant w e akn e ss b e caus e i t do es not provide a m e cms of handl i ng multipl e ocourre noos-ef 
index k e y words in th e transcript materia l? 

vidoo camera so as to st e nographically reoord -a-d oposition whit e simu l tan e ously op e rating -a 
video camera cind record e r, fhis syst e m V» ma i n objo et ivc i s lo provide a system to accurately 
record a deposition, not to e ffici e ntly and precisoly 4e oaie and pres e nt sp e cific passag e s of the 
vkl ootapo d&position. It provid e s a lim i t e d ind e xing gtr uotu fo since tiio strok e r e cord made is 
intended to bo a verbatim transcription of th e vorba Ho 6limony mad e in a courtroom^ rath e r 
than any systematic obs e rvations or ass e ssm e nt of th e^ tfoccodinga in the courtroom. 

For all thes e systems d e scrib e d above, -tho-iiijt er's ability to index not e s or comments 
to the r e cording is entir e ly contro l l e d through t he-indie- ia, or notes^ th e us e r has manually 
enter e d in a docum e nt. For both r e al t 4B=i^a^ post processings those syst e ms r e qu i r e th o-usef 
First to process m e ntally th e r e cord ed-events-a nd then generat e and e nt e r t e xt to or eate-r^es? 

style or accommodating th e not e taking functions t ewa rds more standardiz e d and syst e matic 
observation and ass e ssm e nt applicat i ons. In alUfeese systems, no m e thod i s e xplicitly 
sugg e st e d for hand l ing th e cose o fHrepot it i on or extension of c e rta i n ev e nts or act i vj ti0s4fi-a 
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Existinf> system <i support some forms of indexing of rcQQtrdfed .Mtgials in a variety of 
ways. Euro pean patent appiication publication EP 0 495 612 bv T.ainDainK discloses a 
CQmputer^based note -taking system integrated with an audio or video rec ording system. The 
computer presents a docume m editor style user interface to th e ij<tcr who eUher cr eates a new 
document or retrieve5.aja..exii>tinir document to which the user addy notes as a recording is 
made or p laved via the inie^n-a ted audio or videq system. As thQ.uaCT..eoters each note (mark 
or indicum V the indicum is ad ded to the document an d it is timt ; s^uimped and stored in an 
information-to-time-stamp.indeXvThe time stamps are not visible;. to .the user: they are stored 
with the comnuter's i ntcmal. re presentation of the indica entered bv.ih<^ user. A video-fratTjg 
time stamp ftmction time staniPS„iim^ code data received from the audigor yideo recorder 
md creates a time stamn-to-timCTCOdg index. A browser fimction pcrmlu?. ihe user to retrieve 
SC;Ciions of th e recording using the ir idicii^ directly by selecting the indicia. The browser looks 
m the indicia in the first index to retrieve the lime stamp, and looks up the time <:ode of the 
recording In the second index using the tim e stamp, ptaying th e section of the recordini:; in the 
area indict iied. by ^he time code. EP Q495 6 12 afsn discloses ho w time stamping the indicia 
may he apnlicd to cr eadna topic or key word data. By enterin g new> separate indicia spatially 
near a previously entsired. indicia that is a key word or topic whenever an idea or sneaker or 
topic applies to the previously enter ed indicia, later selection of all of the marks spatially 
associated with a to pjiLmU. result in all sections of the recordin^y indexed by the time stamp s 
of the respective indicia to be replayed, 

U.S. Pat. No 4 , 8 41.387. entitled ''Arrangement for Recording aiid Indexing 
Information" a nd iSSU£dJCLRindfuss. discloses a system for re cordin g infonnation relatinf j; to 
an event on a rccor dinii.inedium. such as aii audio or video tape, and for indexing positions of 
handwritten notationajnade on a touch sensitive device and concerning the event to positions 
on the rec ording, medium in order to allow th e user to i dciitifv portions of the handwritten 
niQMiQ0$..for.>vhieh review of the correlated material on the recorded medju m is desired. In 
the recording mode, the device makes.an. audio rec ording of the event o a„a. Standard cassette 
ta pe. Simultaneously , the ekoromc touehpad senses the p osition of t he.user's han dwritten 
notes on the writing surface, and provide s this informa tion jo. the microprocessorwj^^ 
correlates tbfcjrccord of the positions of the handwritten.notations on each pag e with the 
IKi5ilimijC>Jlhe..re.CQrded information on the a udio ta pe at corresponding instants in tipie^ Real- 
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time constrained coff ciaiion. VQctors representing rectangular areas of the display surface each 

^ cohesive group oXhandwritten text are each comb ined with a lapc position that 
correlates to the instant ULlmfcihe handwritin g within that area began. 

M. 4,425, 5M. issued to Miller discloses a system that co mbines a video tape 
recorder with a com ni itjaLigj^ui^h a manner that these two components each automaticaiiv 
record and dis play where rclai cd infomiation is stored in its own mcchanfiun..as.. well as in its 
counterpart mechanis m., ailo.wina the user to determine the location o f all the co rresponding 
data stored both on vid CjO>..taDe and on a storage medium such as a disl cfcUc, by examining only 
one storage mcdiLim^.Nol*^s. about the recorded event or documatilJ[nay b0 entered onto the 
diskette alono with th o^igomatic entry of the correspondin{> reel numbferand frame number 
of the yjd eo record an d^diske^ie and file address number of the computer sioratre medium. 
Another feature disclosed is thc .c.uD^biUty of the system to enter and dispiay jfae lime and 
date on toth the vjdeo tane agd. disk<^l^te recording mediums as well as on both video monitors 
along with th e data address location information. 

U^S^ PatyNo^ 4,924,387 issucdlo Jeppeson discloses a computerized court reporting 
system which provides for periodicall y annotatin g the Stroke record made byjgiej35£L^ 
court stcnOfcjaphic ma ch ine with a time stamn from a.<iystem cloc k while si muU 
sending a time ^imptoa recording system making a videQ.and audio re cordin g of the 
testimony. The iot^ic of a control system determines au tomalicall Y when to time stamp the 
stroke record and Dcrm its.the .user to trigger a control functi on to annotate the video recording 
with automatic '*on tlxe-jfc.COrd" and '"oft^the recprdf ' messafics_wjtii associated tjme stamps. 

U.S. Pat. No. 5,56 4.005. issued to Weber etal. discloses a system providing 

means for the user to first cre ate a "lime zone" cor related to a r ecording of an event and then 
to tlfixibly enter into the lime z one infonnation relevant to the c^fenL. addressing some of the 
disadvantag es of the systems described above. 

These fcal-lime.data correlation and access sy stems ha ve.similar disadvantages, [njlig 
case of U. S. Pat. No. 5,564.005. the system docs.aot auiomaticallv generate meaning ful 
iPdlca at tbe time the "time zones" are estabJlshQd..by.lhe actions of the user. Rather, 
information about the events ci^rrelatcd. to a time zone is ente red later^ in th e_iorm .Of 
hQ'?^>^ri?.^.P_girQ!>^^- bC ficncrated and entered m real-time^ subicgt.lo the conceptual 

and processiog_s0ccd limitations and variance of a user's, abilities. Therefore, the primar y 
inili.al.]ndcx j^uidc to the meaning of the ev ents withi n a time zone are the temporal 
Characiferistics .of the time zone, augmented al.lhc.lime by the me mory of the user. Tn all 
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aciaiferXace desigri is explicitly su giyested for the case 



of renetition or extension of certai n events or acXiviUes^.inJfafcJ^^ usinfj;.a camnion 

indica for events or activities; there by related but occurring, at ditYerent times, thereby 
facilitating or cnhancincf a user's personal noterUU^ma.J^tyk or. accommodating thg nQte 
taking functi Qn.ljiiL.V^ari ctv of appli cMioni;.. For ail except U >S. Pat. No. 5.564.005, the user's 
ability to index notes to the recordi aids„cmirgjy.cQnti:Qlled.through.the indicia, or notes, the 
iUjiiXJhasj;:^iiii?jne;d..^m^^^ time stamps or position captures are those made at 

the time the notes arfe..cni;ci:i;^^Tho$fe. Systems of entry of a 

aoli^SuiKcfenUy C.OrT0j^ponds .w the time or positions of the recording to provide an 
adequate inde jLM(LlJbtejC!£CjCa3lm^ user to.mentallv process the ongoing real- 

timeJgymt^^dJ^iUl^ generate and enter the note. In the case of LJ.S. 

Pat. No. 4.924.387. thi s provides atLadCQUatC indcxina i^UiJCturit? $ince the stroke record made 
is intended to be a verba tim transcr iptloaQf lhc verbal tesiimony made in a courtroom. In the 
9gji^...9.f.I?..P^ hovyever. vyhere a verbatim transcript iQaM.thg..feVmt.may not bi: the 

note-taker's intention, such an assumption does not allow for the laifecrcal-titnc 
aug mentation of previously entered notes^^^^^ recording, with additional notes related 

tolhe^ previbusly entered notes, since the later^^^^^ time stamped with the timcjhgv 

were entered rather than with the time o f the material to wh ich they relate or are relevant. In 
the case of the system disclosed by Rindfuss. nqtes may be entered later, b^^ later^ntercd 
n!CdCS..wiU. b.e..con:ejated with the. position of later they relate or are 

B;leyant> in the case of the, system.disclosed by Miller^^ addressed to the diskette record 
of ihc cve ntiil.a la to. timti..arc.enterii!d entirely during a post- processing phase and not during 
thfe r^airtime. recording of the events In some of these systems, neither time stamps nor tape 
Di)siiions.are.yi$ibte..lo.lhe user and so the temporal or spatial context of the entered indicia is 
not available to the user to enhance the retrieval ftmction. in addition^ in BP 0 49gj^|2^^^^ 
function provided for creating topic or key words from the entered indicia mav 1^. practically 
limited to one display '"page" or screen unless the us^^^^^ or kev word on a 

second screen or scrolls between screens to add^a^tnj^^^ previously entered topic or ^kcy 
word. As w ith augmentation of no tffAJtL&mctaLihc.ceJ$„nO-fa€U^^ key word 

QrjQpjc„.narac;.Cfcatcd at a.latcr. ttoc.with noics. entered earlier. 

Existing post-processing (non-real-time) anno tation j ^ystems jn the field of p ost- 
producti on video editing provide for the creation of annotations about scenes correlated with 
'*in" and "out" time codes identifying the scenes on a video recording. U.S. Pat. No. 
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5.218.672 is aajgJicmiple of such a system, ft is disclosed there thaLaccne descriptions mav be 
revised after initial crea tion, but the c orrelation of the annotations are confi ndLlj;.^!?^^!!^ to 
the identified scenes. 

In the post-Dro c.essinR.^y$tem tor the corrgiation oQcgd depositi ons; with video 
recordings ther eof disclosed in U.S. Pat. No. 5. 1 72,2RI . a time e(>d.g.nuxnber is assigned by an 
operator of the systCT n. tO both the comnuter transcript and the videoianc. segment where each 
question and answer p assage begins , requiring manuat time cM<^..entrv..:rhe location of 

Lio lfae trianscript may alfjo be correlated with ihcir...COrri0sponding position in 
QrdinK, .but there is no such result during t he rctil-lhue process, in addition, the 
s^QUappeAir. to. provide for the entry of notes or aimaiatic>p$, 
^^Srdesfik eDe^pite the number of such prior art methods, »ft< ^here exists a need 
for ft fta recorded information system that will digitally record events ^ processes, or othor 
obs e rvabl e p henome na, and !i i imuitan e Qus l y, e fficiently, and-aooufatoi y.and aulomaticaHv 
generate intbrmative-Mdexc simliCAthat arc correlated in realtime? with mhrimal interftTence 
te-the recording of the event*; without afTectinp; an intftrview^r^g or observ^ f-Vabilily to lake 
notes or assess the flow of events, Thero4 6 a more gen e ral n ee d tor syst e tnot i ov etruc tufedr^ 
aUmdardiz e d obs e rvation and afjr »e asmo Bt' measurement methods, and th e r e ia q nco d4o 
generate atiUidardiz e d, syst e matic, and quanti tati ve indext s s that ar e l e ss vqubject to i nd i v idual 
variance and which can b e us e d b e t^v ee n and -a mone record e d infoimation p e rtaining to 
djffer^^ft ^dividuals or ev e ntti so that compariso ft^-a ftd other analys e s can b e mad e . 
Fw^ierfHOi%44lt would be desirable to have a system thai allows subsoquont annotat i on ofi for 
annotfttinfi an indexed digital recording of information in such a manner that the annotation 
did not disrupt the standardization of the index or otherwise adversely affect the ability of one 
set of recorded infonnalion in a standardized index key to be compared to another set of 
recorded infomiaiion that us e r » having the same standardized index key. 

Ihereii^ atso a need to gene rate a standard seiof indexes that are not subject to 
individual, variance and which can be used herweea.and among recorded information 
pertaining to di fferent individuals or events so th at comparisons can be made. In the case of 
video and audio jo p^ing tools, £XCJ]USively post-proces^ng^ystetns are.xoad.Cpqu.ate for 
generati ng npte^ aod.JTelatiQnships between recorded seamcnts^and arc time consuming 
be<;^jausc..they require review o f the entire tape i g^oidCT to generate an index. Automatic 
indexing of video and a udio notes b y systertLtimc stamping of key- or handwritten sjxokcs or 
autumatic d etection of speaker voic e chan ges does not provide adequate context markers for 
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the rccor( lfcdL!^iCT»UiLQr..dQ^i> not permit user control of the amounLof detail to be captured. 
For cxamole. an indexj£rg;i»tfed. pn the bas is of speaker sgamentatk^^ would telj 

who wa^js^cakinil-butBO of the talk. F.stahlishinfi timc„zones thro ugh the 

action of 4 uSCr dOCS noLdlrectlv generate indica containinf» information regarding the events 
within th(^Jim^.zone$. 

SUMMARY QF TtfElN VBNTION 

OB JECTS AND ADVAN1 A 6E^ 
Accordingly, it 

U is an object and advantag e of the present HljOve tition to combin e syst e matic^ 
s tandardii£ud obtiervation and ass e ssment us i ng p fi Mlefincd descriptors or ineasur e ment it etas 
wHh-frifi 

4n-FeaMjmc-rA<>d44 ional objects and advantag e s ar e : 

— to-provide-a-syst ctn and method that supports th e g e n e ra tionran^ 

— to provid e a system - and ffl cthod that ctm import and stor e a s e t of prc - d <;^fifted 
descriptor or m e asur e m e nt it e ms, such os O fV OKi s ting questionnair e or s e t of rating scai ^«rfef 
subi^quent us e i n syst e mat i c obs e rvatio ti-mK l ai;so i i;i>mont; 

■ to provid e a syst e m and method- thta t - uses pro - oxisting descriptor or m e asurem ofrt 
ilemi^ 10 t;tructur e th e obs e rvation and ogso ssm ent of cvonta; 

— to provide a system and m e thod tha ^M^g-me a^ ^ urcmcnt items that employ 
quantitativ e rating scal e s; 

— to provid e a system and m e thod -U^ at oflicientiy creates an ind e x for any type 
recording by ful l y automatic oorrekt te n with the use of pre existing d e scriptor or 
measurem e nt it e ms; 

— to provide a sy s tem and method that stor e s r e corded mat e r i al i fhdigit at form in 
atrofti e - s tor age ; 

"-t o provide a system and method that g tefes-deseriptio ns or ratings of observ e d and 






to provid e a sy stem-a nd method that allosvs unlimit e d r e peated u 5eH>^ny 

rmeagur cment item — in other words, an obs e rver ca n-repeat <;dly asse s s 
or rat e a phenomo netH^f-a-f ecurring ev e nt using the sam e measurement i to ffly 



13 



Received from < 7036108668 > at 6/30/03 5:35:57 PM [Eastern Daylight Time] 



r 



7036108668 



MAIL ROOM FAXl 



17:18:32 06-30-2003 



15/54 



Docket No. C}2622^9()8496 
AppL No. 09/539,772 

Preliminary Amendment filed with KCl£ lteque$t 

— to provid e q g ystem-a nd method that a l lows unlim i ted ooncatcnation of i ndo - > fred 
record e d mat e rial wh e n oa e or moro - phcnomena or rBcurring e v e nts ar e repeatedly mifiessc d 

— to provid e a sygtem and m ^s thod that can us e any part or partr ^ of the oontent of 

preexisting descriptor or me asufo mcnt iiams at; ind e x lab e lr. or k e y words; to prov i d o-^ 

syst e m and m e thod in wh i oh a g tat»dft rd ind e x, bas e d upon a sp e cific qu e r i t i onnoiro or 

r e cordings; 

— to provide a s y^i gm and mcihoJ that can s e arch, sort, and/or rc t^4ev<H=^0Ofd<^ 

ft"4idfflS4ise d In the ind e x.; 
— to provid e a syst e m an d-m^h od thai can soarch, sort, and/or retri e v e r e e ofded 
materia l s bas e d upon numerical rati ngs ^om standardiz e d obs e rvation and quantitativ e 

to provide a syst e m and m e thod that po f mits assessors to r e ord e r r e cordings of 

disc e rnible; 

— l»-pfOvldi^ a s ystem and method that allows a as e r, suoh o& a-siodcnt or Irdinee, to 

eftf-^tem- asscssm e nt 








to provide Q gyst efl 

pl^%aek-so4haH=at cr efficiency, accuracy, and r e l i ab i l i t y c an b e-measitfcdi 

upon a s ingle original r e cording, and that Qllo w&^n ultiplc users to playback and rat e at t\\& 
same tim e ; 

— to provfd e-^-syste m and method that supports comparison of ossos sf nent results or 
moiikofing assessm e nt, or oth e r res e arch pui pof ie s; 
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— to provid e a syr i t e m and m e thod that oa» autom ^ ically compare ralings lo make it 
e ajsior to leach, monitor, and conduct r e search; 

— to provide a syst e m and method that allows unlim iteckn^ua l addiiion of indox 
0Oinli? tmd labolii; during playback: and invention to provide a system and method for duiitaily 
rc.cordina.informatio n about an event or individual and aencfatinp; a si^dard jsciLQf-indexes 
liarihis information. 

It is a further ob ject of the in vention to provide a system and meth od. for.annotating 
anJadgxed* ditiital recording of information vi^ithout changing the srandardiz<^;dJnd.t?x 
markings in such a manner that v^ould ady<^rse]y affect the ability of one $et of recorded 
inJfoimation. in g standardized index key to h e com pared to another set olrfecorded 
iiafojmaUoahavinj; the same standardized index jcey^ 

Ici.accotdance..with.a preferred embodiment of the j^ys^ invention, the in vention 
employs ggyjitom .that, includes: (a) means for presenting a subject or or>crator ufith a 
measurem ent item: (b^ qjlCCQrdgr .thai records lu id stores o bservations of events associatcti 
with the m easurement itemJtL.adifiital recording that includes a data t^^^^ sequenced bv a 
time track of time dat^ and ( c) index iojt software that enters an index mark based on said 
time data into an editable database denoting said measurement item. Any annotation, cross 
referencing, or entry of notes vy ill occur in the database iii_a$$OCiatiQn with the time data track 
index without aff ecti ng the index. 

In.us0,.the testing; syst em of the invention includes thcJGailowin>^.sieps:f aVpresenting 
a measurement item. to a subject or operator: (h) difi itallv rc c.oidini£ on a di«;ital storage 
medium at i feast One event ass oci ated with said me asurement item . Mid diidl^irfeCording 
hayinfc.a..event.data.track of event data sequenced bv a time trackof lime.dataif c> indexing 
jtaLd-didtai recording by entering time data related to said event ail<0.a.Jleld.On.Computer 
database. along with an identifier of sajd measurement item;.and.(d) optionallv, augmenting 
said computer database bv adding information to said database, that„relates to said event field 
without chan^in^ the relationship of said time daca to said event data> — to provide a syst e m 



The system and method o f the invention permilmeasuremenl items to be associated 
to observed events, the responses recorded and indexed by measure ment items onto a 
cpmputery b^^ for revi ew, annota tion And, augmentation of diverse information 
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without afte ^:^^^^ ^^^^ se quencing of the recorded dauum the.ubil ity to repla y a portion or 

allofthc recQrO^ tisvents. Such a system will allow assessors and oihm using. ly "test ai^^^^ 
r^$ponse'Vm ethod of i nvestigation much ercatcrnotc-takin g and ann otation flexi bility than 
was preyiously available with prior systems. Tlie system of the ifiy.t^ntion also pennits 
ussessors to recorder recordings of eyents and compare simil;^ Lwe ntg_fiT2mj^ 
Subiects-r on the basis of the associated measuremcnl itcm<; in ordcr to inyestif^ate possible 
relationships and/or trends not previously jjsccmibic. 

Furth e r obj e cts and advantag e s of the lflven tton will becom<3 appar e nt from a 

consid e ration of th e drawlng.s and enGuing -^* scua£ t ion. 



The present inv e ntion combin e s syfit e matio ebsejFvatk>ft- and a^sc ! !;smcnt using 

4tems-witb-stt] 




efiReiefHy^ .fConsi s tency, and a<)curacy in r e tr ie ving and utilizing th e rcoord e d material. Baa e d 

quantitativ e dcscr i pt kfflj^-O f numerical rating reiiu l ts from us e of th e d e flcriptorj? o r 
m e asurem e nt i t e mQ, th e indexes eon re adily be furth e r proc e ss e d and manipulat e d to 
add i tional advantag e . Th e invor rtien-p rovidcs a method for acc e ss and manipulation of di gital 
r e cordings of e v e nts by u.q e of th e se pre - c ntered or pro - CKiaiing mtKttiur e m e nt it e ms as 
meaningfii l index information d e gignatO fiS-s uch as key words, lab e ls^ it e m numb e rs^ or 
num e rical ratings. Such a m e thod is part j ou tarfy- usod for proccsa fe s r e lating to syst e mat l o 
int e rview or ass e ssm e nt m e thods, inolud ing-tfto training and monitoring of int e rvi e w e rs or 
ass e ssors and th e storag e , r e tr ie val, analysig r^fl d - othor manipulation of r e cordings. 

BRIEF DESCRIPTION OF THE DRAWINGS 



t«^ment-i{om-&e 
ssessmeflt4fl-N€- uropsychiatry. 
FlCi. I is a floyL^msiXSimMih^.Xfi&OTdm^ and indexing; functions of the p resent 
invention. 

Fi^JG. 2 is a iis<^n(graliziid block flow diagram illustrating the Gontlgur at k>n of the 
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FigG. 3 A and 3B illiiiitrate further dotaU »e f the syflt e matio aGs e sanitfnl and 

of th e pr e sont inv e ntion.Fig, Ain'd block diagram iltotrating th e configuration of the s<? afehr 
sort. r e tri e vaK of the playback^ and manipulati et^diting fanctions of a pr e ferred ombo dknettt 
of the pr e s e nt inv e ntion, 

Fig> 5 is a block diagrtmi illustrating Ihe ee nfigUfation of post proc e ssiing sy i- ricmatio 
re - assesr , mcnt fiinctions of a preferred ombodinK^nt -e^f-t he present inv e ntion. Hgi). 6A cmd 6B 

embodiment of the present invention. 





HnglM 
rws-speeffiG-detatls 

jo-detaiis- n ee d not b e 



DETAf LED DESCRIPTION 

l^he prtfsont invcnUon i s^anryneans-aBd-m 
index e s for recorded m ater ial^i 

itr^v4U-be-apparen 

used to p ra ctice the pres e nt i nv e ntion. In oth e r inst^mces, well kno wn- struouirer ^ , int e rfac e s, 
^d process e s hav e not b ee n shown in detail in order not ■ t ^ nneoe5r>ar i ly obscur e th e pres e nt 

A pr e f e rr e d e mbodiment of the invention wi lt-n ow bo d e r ^ crlh e d h e r ei n b e low with 
refe reno e to tli e drawings. To e xplain elearly, explan attoft- wi}) b e mad e i n accordanc e with 
th e following ord e r: 

( 1) w i th r e sp e ct to conducting an aSKfeiifemont in ter-viewy 

(2) with resp e ct to playing back an indexed i n t^rv4ewy 

(3) with r e spect to s e arching and r e trieving indexed m aleffetet 

(4 ) with r e sp e ct to r e assessing a prior aasos i iimenl intCfv iewT 

the World Health Organization (-WHQ)-Sehedi ries tor C l inical Assessment in 
defiftrtion-ol^phe nQm e non of int e rest; 2) instruction^^ regard mg-obs€watiOB-^>^^ 
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analysis int e nd e d, Exampl e r i of analysif , include -behavtef al responses to audio and/or vijsuar 
information (qu e stiont;, dir e otionr , , lights, soundg, cto,)> phygi ea l exaniimiliona of laboratory 
apecini e ns, oral d e positionis of witn e oaeg, e tc. The SGAN - f i g >pf OSontii a detailed in s trum e nt for 
clinically bas e d, standardiz e d priyoh i atrio ags e nrimont of psyob o pathology ph e nom e na. Th e 
SCAN contains ov e r l,5QQ m e a.qur e m e nt i t e mcvo aeh- of which i s compos e d of at l e ast an it e rw 
numb e r 102, an item titl e 104, and on i t e m rat i ng -seal^l-26-^Pi g. IB) with rating sca le t e xt 
122 and rating scal e quantitat i v e ratinga 12 4- o ffij- optionally an item d e finition 106, an i t e m 
glossary e ntry 108 providing furth e r deflnitionv ^gge siod probes 110 (qu e stions) for 

^-t^-a^ ^f which combined are r e f e rr e d to as the 

constitu e nt cont e nt may b e cons i d e r e d of y an o bjec^t-^vU h respect to ita function for conducting 

tf^eeus-reeofdingr 

fflea&t^r cment ilemfa 202 to Iho user, whil e simultan e ously proc e ssing an d^ gitally recording 
th e obs e rvable ovo n^kvbe-f aiod, in this cas e th e psychiatric int e rvi e w 20 4 . - As th e-t^se r carri< ^ s 
out systematio o bservatioft- ajid aii^sc^smt^nt with s e l e ction of a m e asur e ment iten> -306? 
optionally gen e rating m e osuremo nt- fcsuitdaut by using th e rating sca le of th e m e armrem efrt 
it e m, th e syst e m automatically d efi nes a £?cgmcnt of recorded mat e rial 20 8 ^ based upon t he 
r e p e at e d 2 1 1 s e l e ction by the user of m easttfefi icnt itcmii? from th e display e d s e t ov e r th e 
cours e of observat i on and assessmen t s The end of the previous s e gm e nt and th e start of the 
n e xt s e gm e nt is d e fin e d wh e n a m e asuremo nt4 tem scicctcd. For th e first segm e nt, no prior 
s e gm e nt e nd i s d e fin e d, and the sta rt-of-th c segment ia defined by th e start of th e digital 
r e cording. Each s e gm e nt is widoxcd to th <i>^^ frently elected m e asur e m e nt it e m that d e fines 
th e start of th e s e gm e nt 2 1 0. The -^Offl bincd i ^ egment d e finition and s e gm e nt ind e xing 

the-backg round without us e r obs e rvation. Th e d i gital 
asurement-resuk-d ata, and index information ar e stor e d in a data b ase-3-2Q^-Tbe 
ind e x i nforma tioft-4hwt-ffl€hid QS the text content 102, 101, 106, 108, 1 10, l-^3r«t^d4-2Qr^ 

used as a common s e t of k e y words, lab e ls, and -quai>tjtati vc attributes to identify ind e x 
e ntries and allow retrievol of divof se-r ^ordcd event data from d i ff e rent subj e cts und e r a 
common fram e work, T 4>e^^rit ing assessment forms a data bas e fil e that can be stored, 
Mw e d ,- o r j megfftt ed into other digital rocords as n e ed e d by the us e r^ 
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Pfguf<as- 3A and - 3B illuslral c further deta i ls of th e 5yr . t e matio agsos i& mcDt and 

presetit invcntio tVr-The-^ys Com providot; th e us e r with a n e ft of pr e detlned measurement ito tfts 
and meanq foF naW gatteft-^t^^ lcciion 302. Simultaneously, a proo e isri of digital r e cording 
ooouFS 30 4 rM Hatt-^ mc as uremcnt itom is sel e ct e d 306, th e syat e m automat i oQ l ly get s^-lewef- 
leve) addrcg tg-mfemm^ie n - such as a timo cod e to id e nt i iy^ th e start of th e reoordtng - segm ent 
S08rftftd4^t tks ihc scgmont au^l addromi with th e current m e afiuroment itcnv JK K 

asseftjt os - tho s o bjoci, optionally chooiiing to us e th e rating r ^ oole from the - W frefi t "measurement 
Heffl-te-geftef^te-measuf^mmH^a i^^ datit 312. Wh e n th e n e xt m e oT i ur e m e nt i tem is sek?eted 

e nd of th e r e cord i ng -segfnH&nt44^r< ^catcs an index item by linking r e cording s e gment s tart 
and end id e ntifiers to t h^mptey cd mcaauromcnt it e m, along with any m e asur e m e nt r e$ulte 
data 320, and th e n stor e s th e i ndeje-item- m the data bitom 220. If th e m e asur e m e nt it e m war. 
pr e viously us e d, and th e r e for th e r e is an cx feti ng index ittgm 321 using that m e asur e m e nt 
it e m, th e n th e r e cording s e gm e nt start and end idc fttifieps -are app<?ndixl 32 8 to th e e xisting 
index item and stor e d in th e data base 220. 

Figure 1 ii> a block diagram illustrating th e contigurat 4eft^f4t ^uory, roiriovaJ, 
manipulation, playback, and indexing functions of a pr e ferred - o m bod i mont of the protr u nt 
mvcntion. One moans of retrieving record e d mat e rial is through use of th e^ re - definod 
measurement itema 102 that were ut;ud in th e cr e ation of th e index i tc m fe-From tho display, 
the-uscf-o an view and select a measurement it e m. Lik e wis e , th e dgtg bas e-ean-b e queried b y 
s earching the index items for k e y words from th e cont e nt of the assoo iat^d- moasuremont 
i tcms^OI. The data base ean also bo queri e d by s e arching for sp e oitl e d - ass essm ent - results 
106. From those Mictions, either singly or combined 4 0 8 , tho $ystO H^mrt omaiicaiIy generat e s a 
qu ery 110 to tho data bas e 220 that r e tri e ves 112 th e assoc i ated s egment-ot^gment^H^f 
recorded materinl for further manag e ment 416. Any sc arc-hnr^ 
using standard moans, and can b e g e n e ra ti z e d to g earefa-aere 
ass e ssm e nts using standard m eaHS;-Retriove d recorded material can b e s e l e ct e d and 
mampiriated4e^otrtput4o-anethep^le m or file, or can b e play e d back with st oodaM 
functions 11 8 . During playback, additional ind e xing can b e perfor fne^rjMi fther indexing is 
desired by th e user 120, the system supports a choic e 4 22 o f^re^i ng and using new i nd e x 
l ab e ls 124 or using e x i sting measur e m e nt i tems 4 2 4r4he- indoxing process 12 8 d e scribed 
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ea rtior asaociaiet; these index oiem e nts with r e cord e d mat e rial, and th e r e sult s arc stored 4 S 0 

F igure 5 is; illustrates th e configuration of por » t procesning sygt e matio re-asgesom eat 
ftmeticH^s o f a pr e ferrod embodiment of the pr e s e nt invention. A re afinesr i ment mod e 502 rfe- 
ass e s sment subject 50^^ and tin e xisiting ass e ssm e nt 506 can b e s e l e ot e d 502^ aHo wteg-tfae 
obsefv ation through playback of th e r e cord e d mat e r i als association with th e exi s t ffig 
as se s sment 506, The set of pre d e fin e d m e asur e ment that wag used to oarry "O u t - thc original 
asses sment is displayed 508> allowing its use for navigation and iieleotion of spoeifie-tttdexed 
s egments of recorded materials. Standard playback functions are al s o - ava il able 512, The 
fflatefials-obscFV ^ through navigation and p l ayback can th e n be re - as3css ed-u5iftg-the-se^ 




ass e sfim e nt, a n e t of pr e- defined mo asw ement items is displayed 602, from which th e us e r oa n 
sel e ct th e curr e nt measur e m e nt it e m 60 4 ; Th eH^ati ng scale text 608 and quantitativ e valu e s 
606 allow syst e matic rat i ng. Dig i tal audio an d-vid eo recording can b e start e d 620, and w i ll b e 
automatically index e d in s e qu e nc e w i th th e selected meos mHMae nt itoma. For playback, th e 
list of meitsurement it e m s us e d in th e ass e s s m e nt to displaye d-6 32j allowing r e trieval of th e 
e orrolated segment of r e cord e d mat e rial^ for playback via a gtonda rd -s et of playback controls 
6^0; Further, the ind e xed record e d mat e rial can b e retrieved - t hf-et>g h text search of all of th e 
ffK>as^ femcnt item, including rating scal e , text 638. Tii e r e corded m ateriat- Chai is played back 
can bo re - assesaed 650.Th e inv e ntion is d e scr i b e d in detQil invention is dc<;cnbed with 
reference to inquiries from an assessor and responses made by an interviewee during an oral 
interview assessment . It will be understood that the description is equally applicable to any 
form of sy s tematic assessment or observation by verbal, visual, audio, or physical method 
(e.g., questions or directions whether presented orally or in tex iprintcd form, menta l or 
physical tests or testing, us e of softwar e or h afdwaferCtc.) and any -form of response by the 
subject, human or other, of observation (verbal, nonverbal, or physical response) with mere 
changes in semantics to describe the particuhir form of test and response. 

A general testing system according to the invention includes the following steps: (a) 
presenting a set of pr e d e fined measurement 4temsitem to a subject or operator for use in 
structuring and oarr>1ng out the observatie iv-a nd assessment of e vents or ph e nom e na of 
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int e r e st; (b) riimiiltanQouRly mokiag a dig it a l - recording of on e or mor e asp e cts of th e obgo fv^ 
ev e nts or phenom e na : (b) makin g a digital record on a digital storage medium xat least one 
event associated with said measurement item^ said di|?itai recording ino l iid i n g record having 
an event track of event data sequenced by a time track of time data; (c) aiitomatioally 
indexing said digital recording b y automat leatiy cnlcring time data and-an identifier relating 
to the curr e nt s e l e ct e d measurement item into a record in fields on a computer date 
fease database : and (d) optionally, augmenting said computer data bas e database by adding 
information tormation to said data ba se databasc that relates to said m e asur e m e nt it e ms or 6a id 
digital r e oordin g event field without changing proviouiily e stablish e d ind e x i ng relation ships 
and data bafi e r e cords the relatipnsh of said time data to said event data . 

An interviewing ass e ssm e nt system according to ihe invention includes a personal 
computer having: (a) an output communication subsystem that allows communication with an 
inlurviewer or interviewee or interviewer so that pf^-dofified-measurement items can be 
presented and i - Hiilcct<xl; (b) an input subsystem for the4nttji^-tewej^ interviewee to respond 
to the mcasuicmcnl item communication s to g e n e rate o^^tsoasm et H data roaulls ; (c) a recording 
subsystem that-^ui tanoou s l y miikcs a digital record of the attdiervideorp hysioiogic, or 
o th er typ esH»fHreeefda blo ovent dau from either or both of th e interv i ewc i%^ft daAilPAil 
<ymrnun [cations as well as the interviewee's b<jhaviors and responses^ with a data track of 
said-event data synchronized ^eh^ a time track of time data; (d) an indexing subs y&tom4hat 
automatically g e n e rat e s corr e lation eotf i os -lfe at dcfinu tim e s e gm e nts of th e digital reoo f^lfeg 
bas e d upon th e re e l e ct i on or presentatioft - ^f succi - ;ssiv e m e asur e m e nt items, and links -fees^ 
tim e s e gm e nts to th e m e asur e m e nt itemSj th ere by creating a m e aningful i nd e x tor th e se 
s e gm e nts of digital r e cording by use of feo^ onlonts of th e corr e lat e d m e asur e m e nt i ten ts-^ 
any quant i tative m e asur e m e nt rating re s u ltSra nd (o) a digital storag e r e tr ie val subsysto m-lhat 
us e s th e m e ar ^ ur e m e nt it e m - bas ed-ifltde x - to rotriove segm e nts of r e cord e d material fo f 
playback, analys i s, e d i ting, annotatin gr-re-Fa ting. or oth e r r e lat e d purpos e s a digital storaftc 
subsystem that allows the diaital record to be ?itored for plaYback jand editino: and (c) a 
computer w ith databas e_&Qft\y.airC..thal can generate a database event index record. basfed On 
said.timc.data.ror each measur ement item. 

I he means for presenting the subject or operator with measurement items can take 
many form% depending on the type of measurement item involved with the assessment. The 
system and its method of use is mi3>_sj;..c;asilv dcscri bed-here in the context of an interview 
between a subject and an interviewer using a computer to display a series of f ncaiiur e mont 
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itefH-questions to the interviewer as a guide for the interview. At th e .sam e tim e ^ ^h ejjbg 
interview is recorded by a audio or video recorder (preferably digital) that inchides a data 
track synchronized by a time track of time data. As the interviewer moves from one question 
fi.e,, m e o^uremont item) to a tv^^ quOiiiliQ n to another via input 4Htato the computer, thw 
Cjuciition switching thg input is iui act that atuses the data bas e ( e ith e r from softwar e or 
hard ware fu nciionii) d iiLiibaS!Ci to record the time data (preferably a stop data bit for termination 
of the preceding ^ime-segmen t of recording and a start data bit for beginning the next time 
segmen t of recording ) and an associated identifier that links th e tim e fi e gm e nt of 
r e cordin gp ertains to the new question (measurement item) , th e r e by g e nerating on mdo^ f-eatfy. 

As those skilled in the art will appreciate, the indexing act ca!i be done in an active 
interview situation, by the interviewer, as questions are posed. The indexing can also be done 
in passive situation, where the operator is observing but not necessarily actively interacting, 
by the operator entering 4aw teaD_itlPUt into the computer from a list of predefined topic items 
(measurement items) during actions by the subject of observation. Either method will 
generate a datarhfts edatabaRe of event data that is indexed by a series of m e asur e m e nt i t e m ■ 
t^ased-identifiers that can be sorted and retrieved for selective retrieval and playback of the 
events. 

Th e pr e f e n- e d Preferably. the software e mbodimen t-of^lnM nvcntion for a personal 
computer will hav e t hat has the capability to: (i) communicatc^ e - dofmed inquiries to the 
interviewee directly or through an interviewer in a systematic manner?j (it) provide aerating 
scal e s or methods associat e d with or part of tli e pr e- deflned ^m^iri ^scalc or method by 
which said inquiries can be rated or appli e d for th e purpos e of des efiW ng s aid intorviowuo;^ 
and fiii) automatically gen e rate nrovide an index in the form of rclfievablc^ index marks 
associated with the pr e d e fin e d inquiries. A personal computer is used to communicate the 
inquiries and at th e sam e tim e t hen to capture and digitally record or control a recording 
device that records the interviewee\s responses to form a digital record of indexed inquiries 
and indexed responses. As thos e sk i ll e d itvtfao-ai^-' 
de-vdie^^trid-beHJsed-tev-pfevtde-t^iese-ftjf] 

hardware that provides functions deacribod above as provid e d by softwar er 

If digitally r e cord e d DurinH later plavback and if responses are to be further indexed 
during l ater playback , the system can also insert additional response index marks and the 
identifiers for the index tnarks| hgin into the digital record so that digital mat e rials are furth e r 
indoxod ■f^-re trieval. For -^^-pi o. in rovicwing an e arlior res ponses are digitajly marked tor 
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retrieval. An assessor may^also insert additional inquiry index marks into the recorded 
i ntervie w, an oflri e r i nor may wish to e mploy pr e d e fined m e arjur e m e nt it e nir> not un e d 
QriginQlly i n int e rvi e w^ to furth e r ans e ss or rat e th e int e rvi e w. In addition to us e of pre defin ed 
m e asurem e nt it e ms which provide m e aningful t e xt cont e nt that s e rves an ind e x lab e ls or key 
wordsi n e w index Jndex marks can he placed at any point in the recording with manual eatfy 
of a lab e l or k e y words> Furth e rmor e ^ if th e us e r wish es tor example to study or analyz e 
communioation and response patt e rns, n e w measurement it e m s could bo created and s to red 
for r e p e at e d use. Pr e f e rably, an " i nquiry"' tiieosur e ment item could be cr e ated, stored.; and 
then insGrtcdateaohlooat j on of int e r e stbutan inauirv index m ark is pref erably inserted 
where a specified data item inquiry is being communicated, and a -response " m e asuremen t 
item could be creat e d., stor e d, oiid Index m^^^^^^^ inserted where the response is made.. The 
inquiry and response index marks are desirably standardized and relate to the syr i t e matio 
content of the inquiry or the response. 

The invention is conveniently explained in the context of an interview under the 
WHO Schedules for Clinical Assessment in Neuropsychiatry (SCAN). These schedules 
represent a detailed instrument for clinically-teiscd, st^indardizcd psychiatric assessment of 
psychopalhology phenomena. The SCAN ooniains ov^gr 1,500 mtia^furememl it e ms, each of 

>-"^&0fttent^ f a SCAN ly ^easurgmefit-k^^mrAa-examfite^no^u ^ item 
from th e SCAN is shown in Figs, I A and IB , The SCAN is a widely recognized and used 
international standardrhavjftg-beeft-tfaiWJ)ated4ftto-ov©^ It will be understood by 

those skilled in the art that the techniques applied to a SCAN assessment would be applicable 
to other types of assessment instruments, physical examinations, or other physical 
phenomenon whose analysis is aided with a recording of sights, sounds,.phy5iologyriind/or 
other domains of observation with only minor semantic differences that describe the same 
essential types of events. 

Using th e pr e s e nt inv e ntion to conduct a SCAN psychiatric During an assessment 
interview with a human subject^ real-time selection byof the i nt e rv ie w e r of a relevant SCAN 
item in the natural course of the ifrteryiewr<>bseFv atk)n^ and ass essmenfr-of^infeaJ 
0heitomefl ac.veiits automatically generates a scries of time-toe 4siamped inciuiry index eftlFy 
liftfangiks that allows us e of the SCAN item serve i\s an informative indicum for the 
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riimultaneour i ly recorded dat a ( e >g., any of audio, vid e o, physiologic) . I'he tinie-slamp index 
marks (time stamp plus identitler) are stored in data ba&e databajjO fields with time data that is 
correlated to a sequence of event data in the recording of the events. The resultant set of time 
stamp index marks then provides a general index structure which can be used to further 
support, independently of the real-time process, a flexible note-taking stniclure and user 
interface that complements diverse personal note-taking styles and application needs. 

"Notations and annotations in the form of text sounds video, or other data can be 
entered into the data bais e datahase as separate fields of information without uflccling the 
relationship between the time stamp index mark and the recorded data associated therewith. 
Thus, the iLse of digital index marks of an event recording in an editable data bai i c dalab^*^ 
provides assessoi^ with the ability to generate assessment records rich in diverse contcni. 
When coupled with standardized assessment inquiries (e.g., a list of standard interview 
questions) and a data bas e datqbase field that identifies each kiquiry by a standardized 
descriptor for each inquiry, this rich. assessment record can be used as a basis for 
comprehensive comparisons on a.rieh,.. divefse content. 

In the Ciise of assessment interviews in which there is no pre-existing set of 
titandardized m easurement items (questions, definitions, inquiries, instructions^ directions, or 
other tasks), an important feature of the present invention is the ability tor the assessor or 
another individual ( c.gvy an interview designer) to generate a set of items prior to the event to 
be observed, oss e r i ij e d, and recorded . This supports th e proc e ss of dev e lopin g and develop a 
new set of-s tandardized inquiries tor investigating new issues. This set of items can be used 
and re — ^used as real-time index information designators which can later be used as standard 
access meclia^^isms into both the notes and the correlated recorded signals. These new 
inquiries, e.g,, key words, labels, or any other pointer to information, can identify specific 
concepts in the user-produced information tafor which the user wants to have later access. An 
information desigtmtor is treated as an object that can b ejs easily created and reusedT44H5j|,nd 
this novel implctnentation both in the user interface and data structure efor the present 
invention provides a flexible and powerftjl indexing tool ft tf into the recorded signals in 
addition to the access provided by the recorded medium's address parametersparametcr. 

In an environment where a recording of events is being made simultaneously-wUfc-a 
tbe-faiing-el^b osc eventrs the interviewer has limitation s, du e to l i mitations of human 
cognitive abil i li e&y on what actions can be taken or what-additionai items ean-be-considered as 
the interview is in progress. The present invention produces a digital data base datahase of 
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indexed inquiries and responses that can be used to organize the assessor's notesy originally 
written in free text or graphical fomi? but later entered into the data baq e^jafebase with 
minimal restriction as to the amount of that infomiation or when it is entered. The inquiry 
and/or response index marks allow the assessor's notes to be correlated with the recorded 
signals of the event. Further, the present invention provides the assessor witll the ability to 
modify any notes that were previously entered without regard to when the modifications are 
made or where they are located on the recording medium. 

The invention recognizes, moreover, that the temporal or spatial sequencing of 
information is itself often a significant clue as-to the content of the information and may aid a 
user in recalling memories or in rcUicving recorded information from a perceived event. The 
inquiry and response index marks of the present invention provide a mechanism for the user 
to control the playback sequence and thereby view the index with items sorted in, e.g,, 
chronological order, type of Inquiry (as denoted by the type of inquiry index encoded into the 
digital record), by type of response (also as denoted by the type of response index mark 
encoded into the digital record), as well as any other order desired by the assessor 

Tlic system of the present invention is particularly cflcclivc for video and audio 
interviews where the events include a time sequence of responses by the int e rvi e w e e or test 
subject. Tliese events arc recorded as a series of audio or video input data st^gm e nta 
sequenced by a time track of time data. Because the amount of detail that may be captured 
for a particular time on an associated recorded medium is open-ended, a richer set of 
annotations may be effietent ly matia by us e of pr e defin e d meosure meflt-tto mmadc while the 
video is being digitally recorded in real- time than is typically allowed in previous analog 
systems. Tli i s reli e v e s th e-net e - takor (i.e. th e m e asur e m e nt asij e ssof ^ ^many of Iht^ pr e ssur e s 
typically asrioc i at e d with tok te fi int<^rvit?w not e s in r e al tim e . Th e time s e cmeao elhg lime zone 
and reusable information designator object features provide for treating the event as a whole 
and n e rmit hv permitting later augmentation or modification of previously made notes/This 
rjBl ieyes the note-taker-0£matiy of the pressures typically associated with bdeing interview 
notes in real time. 

It is another significant advantage that the system of the present invention operates in 
both real-time and post-processing environments. This feature recognizes that 



nQt e takin g note-takigfi as a human activity is typically an interactive process. The output data 

structure (e.g,, the i0d©?t^video mat e rialsM og). produced by the system of the present 
invention during a rcal-lime note-taking session while the event is being recorded may be 
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further modified and/or augmented by the same user or by different users during post- 
processing review sessions occurring in parallel or get4atsenatin_i> Tliis facilitates and 
potentially reduees the time roquired lnvolved in the.post-processing siessfenssgSMgn. 

A system for correlating event data being recorded in real time to event information 
entered into the system by a system user includes input circuitry connected to a user input 
device for producing signals indicating actions of a system user; output circuitry connected to 
a display having area for presenting images; and a processor connected for receiving the 
signals from the input circuitry, and connected for providing images to the output circuitry. 
The system also includes a digital recording device in communication with a digititl storage 
medium for producing a plurality of signals indicating recorded event data about an event for 
storage on the storage medium. The recording device is currently producing the plurality of 
signals and indicating the recorded event data and is connected for simultaneously prc»viding 
address data items to the processor of the $ystem ..and. to. the recording device . The recording 
device stores the plurality of signals indicating the recorded event data on the storage medium 
so that a recording address value of each address data item provided by the address source 
indicates a portion of the recorded event data. Tlie fiygtem a ystems also includes memory for 
storing data^ including instruction data that indicat e jndjcatij}g instructions the processor 
cx.ecutes. 

Stated another way, the system may be configured to operate in a post-processing 
environment so that an event that has already been recorded but not indexed is the event 
about which a log is convStructed according to the present invention, Tlie event that has 
already been recorded is digitally stored on a storage medium as a plurality of signals 
indiciiting recorded event data about the event, in this aspect of the invention, the system 
includes an output device for presenting the previously recorded signals thereon to the system 
user. This output device is in communication with the storage medium including the plurality 
of signals indicating recorded event data about the event. The output device is connected for 
providing a plurality of address data items to the processor of the system as user-requested 
address values; each address data item is a recording address value indicating a portion of the 
recorded event data. Preferably, the plurality of signals indicating the recorded event data 
includes the plurality of address data items, but the output device may itself generate them as 
the plurality of signals is being presented by the output device. 

The system of the present invention pennits a system user to interactively produce 
stored information that is correlated with an '^address". The "address" refers to a 
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sociutsnoe measurement dimension used to identify a portion of the stored information. When 
the stored information is being correlated with recorded signals, the address typically refers to 
the location of the recorded signals on the storage medium storing the recorded signals. 
However, the system of the present information is not limited to correlating information with 
recorded signals, and may be used in any situation where later access to the stored 
information would be facilitated by correlation with an address, and by having an access 
mechiuiism as provided by the "information designators". Generally, when the system user is 
producing a data bas e d atabase of stored Information, the address record will be time data 
taken from the time track of the recording of the events. 

The computer-controlled system of the present invention uses common, readily 
available hardware components. For an assessment interview delivered via computer^ the 
system includes a computer with an image display that will display questions or directions, an 
input that receives signals from a user interaction device, e.g., keyboard, digital tablet 
microphone, camcr% or other recording device that produces a data track synchronized by a 
time track. The display can present the mear»ur e ment item questions directly to the subject or, 
alternatively, can display a series of questions to an interviewer/assessor who then presents 
the questions to the subject. Some act taken to move from one question to another generates 
the data base database time entry or start/stop time entries as well as the index identifier that 
pertains to the question. 

Analog recording devices can ha v<? their recordingiij ^ digitize d, or the obse rva^ 
evcntii can b e ^ r recorded directly as a digital recording that includes: (a) a data track of 
sound, image, or other event data that arc synchronized by (b) a time track that includes time 
data correlated with event data. The use and existence of these two data tracks is typically 
transparent to the user of the equipment but can be read by a computer and used to generate 
data bas e database entries for each recorded event based on the time track data associated 
therewith from the digital recording. 

The user interaction devices useful in the present invention may include any one of a 
variety of input devices controllable by a human user. For example, the user interaction 
device may include a pointing and gesturing device to indicate actions, gestures, and strokes 
such as those produced when making handwriting gestures. A stylus or other form of 
handheld pen are examples of a suitable user imemction device. Other pointing devices such 
as a mouse , traGkball, e y e posit i on, OF - tH^ain activity el e ctrical or magnctio pa ttefn£> may also 
b e suitabl e tor indtcotma aotiott S :^Ofltur<as. and strok e s, or a trackball may also be suitable for 
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The prcs;ent in veation iii furthe r understood bv refe rence to FTrf. _l.j^hifch„iihows a 
flow diag ram of the .rfcc„QrtiinH and indexing fanctions of the prgjacQLiaMfenUon.. A distal 
recordinfy 1 of a scries olgyeuts is m ade directly bv a digital recorder o r jgaadfe inta a. digital 
formation Z,irim..ai3i..analo«: recording^ The indexing svRtcm 3 is uirn^ to generate a database 4 
of index i tems 5 ba $.cd . On measurement items 6. 

Suilabk„m<i!a^urement items cafi vary over a wide ran^c and will depend on the 
information soup[ht fr om, the test subject and the type of analysis intfendgd.. Examples of 
analysis l ududfe behavorial responses to audio and/or visual information ruuestions. 
ions. lig hts^ sounds.. etc), physical examinations of laboraii>ry- Specimens, oral 

LQ£-Wimcss.feS, elc. in the case of a behavioral exam igatioiX, measurement it^ms 6 
can be generated in oroccsii^ IQ irom a standard assessment instrument .T^ -OPCn. ended 
Questions 8 and anv other inputs 9 desired by the assesso r- Preferably , process 10 is a simple 
system that is readily repeated. Suiteble examples include a question outline or list. 

Similarly. FIG. 2 shows ailQw dia^q-am of the playback a^^^^^ editing function of the 
present invention. Digital data 20 made of an event sequence tra ck of event data is 
synchronized bv an associated time track of Lime data.is played back 2 1 for ai^^ 
and/or ph ysiclot;ical events. Preferably, new index itemj£.22. are. create d during a post;^ 
iarOi?j;?S$injj:. playback and revie^y phasejuit may alsti,b!£_cjcated. separa t ely^ for example as a 
prc-cxistini LdgUiiM set of me asurement sub-items, or as a^ldilionai notes during the orig inaj 
a<!scssme nLDrQcess. _Each index items is base d on a link 2 3_.bfetween the time data in the time 
track with corresponding event s equence data and entered as index da ta 24 in a database. This 
database c an^ jnclude associated records that identify the index entries by a_.commQn set of 
labels, index attribi Ucs 25 Qr_Qther identifier that allow diverafe.C-Vent data from different 
S.uMC!ClS under a common framework. The rcsultin^ ^ a. s$.^$jjment forms a database file 26 that 
£arLb<lJ^lOrcd. shared, o r integrated into other dif : it 4Lrg.C.0rds.a$. needed by the assessor. 

Although the foregoing disclosure relates to preferred embodiments of the invention, 
it is understood that these details have been given for the purposes of clarification only. 
Various changes and modifications of the invention will be apparent, to one having ordinary 
skill in the art, without departing from the spirit and scope of the invention. 
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